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Abstract 

Automatic temporal ordering of events described in discourse has been of great interest in recent years. Event orderings are 
conveyed in text via various linguistic mechanisms including the use of expressions such as "before", "after" or "during" 
that explicitly assert a temporal relation - temporal signals. In this paper, we investigate the role of temporal signals in 
temporal relation extraction and provide a quantitative analysis of these expressions in the TimeBank annotated corpus. 
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1. Introduction 

The task of automatically determining the tempo- 
ral relations that hold between events described 
in a text is a research challenge that has in- 
creasingly occupied researchers in computational 
language processing ( jSetzer and Gaizauskas, 20001 



Pustejovsky et al, 2004] |Verhagen et al, 2009 

Verhagen et al, 20T0) . The mechanisms used to 
convey temporal relational information in text are 
complex and include tense, textual ordering, as well as 
specific lexical cues; and of course readers and writers 
bring to bear lexical and world knowledge, informing 
them of likely event sequences and inter-relationships. 
Of the mechanisms that play a part in conveying 
temporal relational information, one that has been 
under-investigated is the use of expressions, typi- 
cally adverbials or conjunctions, which overtly sig- 
nal temporal relations - words or phrases such as 
after, during and as soon as. Very few of the 
teams participating in the recent TempEval chal- 
lenges ( [Verhagen et al, 20091 |Verhagen et al, 2010 1 
exploited these words as features in their automated 
temporal relation classification systems. Certainly no 
detailed study of these words and their potential con- 
tribution to the task of temporal relation detection has 
been carried out to date, despite their demonstrable 



utility (Derczynski and Gaizauskas, 2010b I. This pa- 
per begins to address this deficiency. Using the Time- 
Bank corpus, a corpus of news wire texts annotated 
with TimeML ( [Pustejovsky et al.,~2 003), in which a 
class of expressions referred to as temporal signals is 
explicitly annotated, we set out to answer the follow- 
ing questions: 

1. What proportion of temporal relations annotated 
in TimeBank have an associated temporal sig- 
nal? That is, are explicitly signalled using a sig- 
nal word or phrase? 



2. Of the expressions which can function as tempo- 
ral signals, what proportion of their usage in the 
TimeBank corpus is as a temporal signal? E.g. 
how ambiguous are these expressions in terms of 
their role as temporal signals? 

3. Of the occurrences of these expressions as tem- 
poral signals, how ambiguous are they with re- 
spect to the temporal relation they convey? 

The following paper provides provisional answers to 
these questions - provisional as one of the difficul- 
ties we encountered was significant under-annotation 
of temporal signals in TimeBank. We have addressed 
this to some extent, but more work remains to be done. 
Nonetheless we believe the current study provides im- 
portant insights into the behaviour of temporal signals 
and how they may be exploited by computational sys- 
tems carrying out the temporal relation detection task. 
The remainder of the paper is divided into three parts. 
In section two we give a more detailed characterisa- 
tion of temporal signals, further describe TimeBank 
and TimeML and discuss prior related work. In sec- 
tion three we describe the additional annotation work 
we have done on TimeBank and present the quanti- 
tative analysis that provides answers to the questions 
framed above. The fourth section considers, on a case 
by case basis, specific examples of expressions which 
are highly ambiguous as regards their role as temporal 
signals and discusses their behaviour in detail. 

2. Temporal Signals 

2.1. Linguistic Characterisation 

Signal expressions explicitly indicate the existence 
and nature of a temporal relation between two events 
or states or between an event or state and a time point 
or interval. Hence a temporal signal has two argu- 
ments, which are the temporal "entities" that are re- 
lated. One of these arguments may be deictic instead 



of directly attached to an event or time; anaphoric tem- 
poral references are also permitted. For example, the 
temporal function and arguments of after in He slept 
after a long day at work are clear and available in 
the immediately surrounding text. With After that, he 
swiftly finished his meal and left we must look back to 
the antecedent of that to locate the second argument. 
Sometimes a signal will appear to be missing an ar- 
gument; for example, sentence-initial signals with 
only one event in the sentence {"Later, they sub- 
sided."). These relate an event in their sentence with 
the discourse's current temporal focus - for exam- 
ple, document creation time, the previous sentence's 
main event, or reference time ( |Reichenbach, 1947 



Dowty, 1979 1. In a more complex case, such as Exam- 
ple [Q we suggest that two temporal links are present. 
First, Later is attached to the current focus, as is sur- 
veyed. Secondly, after describes the relation between 
the storm and surveyed. 

(1) It rained heavily. Later, after the storm, we 
surveyed the damage. 

Sometimes a signal may appear to only take one ar- 
gument, when the other is (implicitly) reference time. 
For example, afterwards and after that are temporally 
equivalent, though afterwards only takes one extra ar- 
gument. 

Signal surface forms have a compound structure con- 
sisting of a head and an optional qualifier. The head 
describes the temporal operation of the signal phrase 
and the qualifier modifies or clarifies this operation. 
An example of an unqualified signal expression is af- 
ter, which provides information about the nature of 
a temporal link, but does not say anything about the 
absolute or relative magnitude of the temporal separa- 
tion of its arguments. We can elaborate on this with 
phrases which give qualitative information about the 
relative size of temporal separation between events 
(such as very shortly after), or which give a specific 
separation between events using a duration as a modi- 
fying phrase (e.g. two weeks after). 

2.2. TimeML and TimeBank 



TimeML (Pustejovsky et al., 2004 1 is a temporal an- 
notation language. It may be used to annotate 
events, time expressions or timex's (times, dates, du- 
rations), temporal relations between events and times 
(such as before or during), and signal expressions - 
words or phrases (such as conjunctions, adverbials) 
that provide information about temporal relations. 



tions, as well as events, times and signals. It consists 
of around 65 000 tokens of English newswire text. 
TimeML offers the following definition of temporal 
signal. From the annotation guideline^: 

A signal is a textual element that makes ex- 
plicit the relation holding between two en- 
tities (timex and event, timex and timex, or 
event and event). Signals are generally: 

• Temporal prepositions: on, in, at, from, 
to, before, after, during, etc. 

• Temporal conjunctions: before, after, 
while, when, etc. 

• Prepositions signaling modality: to. 

• Special characters: "-" and "/", in 
temporal expressions denoting ranges. 

In cases where a specific duration occurs as part of 
a complex qualifier-head temporal signal, e.g. two 
weeks after, TimeBank has followed the convention 
that the signal head alone is annotated as a signal and 
the qualifier is annotated as a TIMEX of type DURA- 
TION. 

2.3. Previous Work 

Signals help create well-structured discourse. Tem- 
poral signals can provide context shifts and or- 
derings ( |Hitzeman, 1997| ). These signal expres- 
sions therefore work as discourse segmentation mark- 
ers (Ho- Dac and Pery-Woodley, 2008) ). It has been 
shown that correctly including such explicit mark- 
ers make texts easier for human readers to pro- 



cess (Bestgen and Vonk, 1999 ). 



TimeBank (Pustejovsky et al., 2003 ) is currently the 
largest TimeML-annotated gold standard corpus avail- 
able, including over 6 000 temporal relation annota- 



Some prior work has approached linguistic charac- 
terisation of signals. Bree et al. (1986) performed a 
study of temporal conjunctions and prepositions and 
suggested rules for discriminating temporal from non- 
temporal uses of signal expressions that fall into these 
classes. However, this work is purely theoretical and 
not a corpus-based study. Schluter (1200 It identi- 
fies signal expressions used with the present perfect 
and compares their frequency in British and US En- 
glish. Vlach (119931 ) presents a semantic framework 
that deals with duratives when used as signal modifiers 
(see Section l27TTb . Bree et al. ( 119931 ) later describe the 
ambiguity of nine temporal prepositions in terms of 
their roles as temporal signals. Our work differs from 
the literature in that is it the first to be based on gold 
standard annotations of temporal semantics and that it 

1 See http://timeml.org/site/publications/timeMLdocs 
/annguide_l .2. 1 .pdf . 



Annotated S I GNAL elements 


758 


Signals used by a TLINK 
Signals used by an ALINK 
Signals used by a SLINK 


721 

1 

39 


TLINKs that use a SIGNAL 

Signals used by more than one TLINK 


787 
54 



Table 1: How <SIGNAL> elements are used in Time- 
Bank. 



TLINKs per signal 


Number of signals 


1 

2 
3 
5 


597 

41 

12 

1 



Table 2: The number of TLINKs associated with each 
temporal signal word/phrase, in TimeBank. Signals 
not used on TLINKs (e.g. those used on aspectual 
or subordinate links, or for event cardinality) are ex- 
cluded. The distribution is Zipfian. 



encompasses all temporal signal expressions, not just 
those of a particular grammatical class. 
Intuitively, signal expressions contain temporal order- 
ing information that human readers can access eas- 
ily. Once temporal conjunctions are identified, ex- 
isting semantic formalisms may be applied to dis- 



course semantics ( |Dowty, 1979 1. It is however am- 
biguous which temporal expression they attempt to 
convey (Hitzema n, 2005 [ ). Our work quantifies this 
ambiguity for a subset of expressions. 
Previous work applying temporal signals 
has been related to the labeling of temporal 
links ( |Min et al., 2007[ ) and question answer- 
ing ([Pustejovsky et al, 2005} |Saquete et al, 2009]). 



In particular, Lapata and Lascarides (12006 l i remove 
the temporal signal from sentences containing 
two temporally connected clauses and attempt 
to learn sentence-level temporal relations using 
the orderings suggested by the removed signal as 
training data. Directly applying signals to the tem- 
poral relation identification task, Derczynski and 
Gaizauskas (2010b) halved the error rate of TLINK 
classification for TLINKs that have a signal by adding 
features describing signals. This raised classification 
accuracy from 62% to 82%. 

3. Signals in TimeBank 

In this section, we give a detailed profiling 
of temporal signals in the TimeBank cor- 
pus. Statistics are generated using the CA- 
VaT ( [Derczynski and Gaizauskas, 2010a l tool 
for TimeML-annotated corpus analysis. 
First, we note that in TimeML signals may be divided 
into three classes based on the type of relation they 
signal: temporal (tlink), sub-ordinating (slink) or as- 
pectual (alink). The distribution of signals by class 
in Timebank is shown in Table [TJ For the rest of the 
paper we discuss temporal signals only. 



Part of speech 


Frequency 


Proportion 


IN 


521 


77.3% 


RB 


73 


10.8% 


WRB 


53 


7.9% 


JJ 


14 


2.1% 


RBR 


5 


0.7% 


VBG 


4 


0.6% 


CC 


2 


0.3% 


RP 


1 


0.1% 


JJR 


1 


0.1% 



Table 3: Distribution of part-of-speech in signals and 
the first word of multiword signals, using the Penn 
Treebank tag set. 



3.1. Additional Annotation 

Upon examination of the non-annotated instances of 
words that often occur as a temporal signal (such as 
after) it became evident that TimeBank's signals are 
under-annotated. As we are certain of some annota- 
tion errors in the source data, we revisited the orig- 
inal annotations. A subset of signal words was se- 
lected for re-annotation. This set consisted of sig- 
nals that were ambiguous (occurred temporally close 
to 50% of the time) or that we expected contained, 
based on informal observations, would yield a num- 
ber of missed temporal annotations. All tempo- 
ral instances of these words were re-annotated with 
TimeML, adding EVENTS, TIMEX3s and TLINKs 
where necessary to create a signalled TLINK. 
A single annotator checked the source documents and 
annotated 70 extra signals, as well as adding 34 events, 
1 temporal expression and 49 extra temporal links. 



Expression 


Count in corpus 


As signal 


Proportion as signals 


After curation 


Proportion 


in 


1214 


161 


13.3% 






after 


72 


56 


77.8% 


66 


91.7% 


for 


621 


52 


8.4% 






if 


65 


37 


56.9% 






when 


62 


35 


56.5% 


56 


90.3% 


on 


344 


33 


9.6% 






until 


36 


25 


69.4% 


36 


100.0% 


before 


33 


23 


69.7% 


30 


90.9% 


by 


356 


20 


5.6% 






from 


366 


19 


5.2% 






since 


31 


17 


54.8% 


18 


58.1% 


through 


69 


15 


21.7% 






as 


271 


14 


5.2% 






over 


59 


14 


23.7% 






already 


32 


13 


40.6% 


13 


40.6% 


ended 


21 


13 


61.9% 






during 


19 


13 


68.4% 






at 


311 


11 


3.5% 






previously 


19 


11 


57.9% 


16 


84.2% 


within 


23 


8 


34.8% 






s 


10 


8 


80.0% 






later 


15 


7 


46.7% 






earlier 


50 


6 


12.0% 






while 


39 


6 


15.4% 


9 


23.1% 


then 


23 


5 


21.7% 






once 


15 


5 


33.3% 






still 


35 


4 


11.4% 






following 


15 


4 


26.7% 






meanwhile 


14 


4 


28.6% 


9 


64.3% 


at the same time 


6 


4 


66.7% 






to 


1600 


3 


0.2% 






into 


63 


3 


4.8% 






follows 


4 


3 


75.0% 






subsequently 


3 


3 


100.0% 






followed 


10 


2 


20.0% 


4 


40.0% 


former 


16 





0.0% 


12 


75.0% 



Table 4: Frequency of candidate signal expressions in TimeBank. We include counts of how often these occur 
as signal expressions both before and after manual curation. 



3.2. Proportion of Temporal Relations with 
Signals 

TimeBank contains 6 418 TLINKs (6 467 after re- 
annotation) of which 718 (787) are explicitly indicated 
by a temporal signal - 11.2% (12.2%). This provides 
an answer to the first question we posed in Section 1. 
Thus while ability to successfully detect temporal sig- 
nals will not solve the problem of assigning temporal 
relations, it is likely to make a noticeable difference 
(see Derczynski and Gaizauskas (I2010bl> ). Perhaps of 
more interest is that so few temporal relations are ex- 
plicitly signalled - we must look elsewhere for an ex- 



planations of how temporal relations are conveyed in 
natural language. 

While many TLINKs do not have any associated tem- 
poral signal it is also the case that some temporal sig- 
nals are associated with more than one TLINK. Ta- 
ble [2] shows details of just how signals are being used 
by TLINKs. 

3.3. Temporal vs Non-temporal Uses 

The semantic function that a temporal signal expres- 
sion performs is that of relating two temporal entities. 
However, the words that can function as temporal sig- 



Signal Expression 


TLINK count 


w 

H 

PL. 


w 

o 

W 
03 


z 

o 
w 

M 


I* 

pq 
i 

Z 

£3 

o 

w 
ffl 


o 
z 

2 

|D 
Q 


I* 

PQ 
I 

Q 

a 
Q 

Z 

w 


Q 
Z 
W 


B4 

pJ 
H 

Ph 

< 


O 

Ph 

w 


V) 

W 

Q 

£3 

U 

Z 


Q 

— 
Q 

£3 

u 

z 

l— 1 
l/J 


C/5 

ID 
O 

pq 
Z 

1 

E3 


after 

when 

until 

before 

since 

already 

previously 

while 

meanwhile 

followed 

former 


76 
57 
37 
36 
19 
13 
18 
9 
9 
4 
12 


62 

16 

4 

1 

9 

6 

2 


3 
3 
7 

28 
1 
6 

12 

1 

2 
12 


4 
1 
1 

2 
2 


2 
7 


1 

2 


21 
1 


5 

1 

2 


2 
1 
1 
1 


1 

2 
1 


9 

4 


9 

3 
1 


14 

9 

5 



Table 5: Signal expressions and the TimeML relations that they can denote, ordered as per Table 4 for compari- 
son. Counts do not match because a single signal expression can support more than one temporal link. 



nals also play other roles. 

Table @] details the distribution of expressions that are 
found as temporal signals more than twice (after re- 
annotation) in TimeBank. The most frequent signal 
word was "in", accounting for 24.8% of all signal- 
using TLINKs. However, only 13.3% of occurrences 
of the word "in" have a temporal sense. The word "af- 
ter" is far more likely (91.7% of all occurrences) to 
have a temporal sense. In total TimeBank contains 62 
unique signal words and phrases (ignoring case) and 
of these, over half (36) are also found in Table |4]. 
As an aside, note that any thought that temporal sig- 
nals might be easily picked out based on word class 
may be dispelled by examining the distribution of 
parts-of-speech possessed by temporal signals - see 
Table El 

3.4. Relation Ambiguity 

The nature of the temporal relation described by a sig- 
nal is not constant, though each signal tends to de- 
scribe a particular relation type most often. Table [5] 
gives an excerpt of data showing which temporal rela- 
tions are made explicit by each signal expression. The 
variation in relation type associated with a signal is not 
as great as it might appear as the assignment of tempo- 
ral relation type has an element of arbitrariness - one 
may choose to annotate a BEFORE or AFTER relation 
for the same event pair by simply reversing the tempo- 
ral link's argument order, for example. Nevertheless, 
it is possible to draw useful information from the ta- 
ble; for example, one can see that meanwhile is much 



more likely to suggest some sort of temporal overlap 
between events than an ordering where arguments oc- 
cur discretely. 

4. Per-expression details 

We chose to curate signal annotations in TimeBank for 
a subset of candidate signal expressions (as described 
in Section l3~T! . During this curation, we attempted to 
determine distinguishing features that could aid auto- 
matic discrimination of temporal from non-temporal 
sense of the expressions. Details of our findings are 
given below. 

Previously TimeBank contains eight instances of 
the word that were not annotated as a signal. Of these, 
all were being used as temporal signals. The word 
only takes one event or time as its direct argument, 
which is placed temporally before an event or time that 
is in focus. For example: 

"X reported a third- quarter loss, citing a previously 
announced capital restructuring program " 
In this sentence, the second argument of previously 
is "announced", which is temporally situated before 
its first argument {"reported"). When previously oc- 
curs at the top of a section, the temporal element that 
has focus is either document creation time or, if one 
has been specified in previous discourse, the time cur- 
rently in focus. 

After Of the nineteen instances of this word not an- 
notated as temporal, only three were actually non- 
temporal. The cases that were non-temporal were a 
different sense of the word. The temporal signals 



pp 



IN NP 

before Det NNS 

I 
the wars 



NX 



Figure 1 : An example of the common syntactic sur- 
roundings of a before signal. 




NX CC NX 

NN and jj NN 

f 'h ' ' 

rounder former chairman 

Figure 2: Example of a non-annotated signal {former) 
from TimeBank's ws j_0 7 7 8 . tml . 



are adverbial, with a temporal function. Two non- 
temporal cases used a positional sense. The last case 
was in a phrasal verb to go after; "whether we would 
go after attorney's fees" . 

When There are 35 annotated and 27 non-annotated 
occurrences of this phrase. It indicates either an over- 
lap between intervals, or a point relation that matches 
an interval's start. Twenty-three of the twenty-seven 
non-annotated occurrences are used as temporal sig- 
nals. Two of the remaining four are in negated phrases 
and not used to link an interval pair, for example, "did 
not say when the reported attempt occurred" . The 
other two are used in context setting phrases, e.g. "we 
think he is someone who is capable of rational judge- 
ments when it comes to power" , which are not tempo- 
ral in nature. 

While The cases of while that have not been anno- 
tated as a signal - the majority class, 33 to 6 - are 
often used in a contrastive sense. This does suggest 
that the connected events have some overlap, often 
between statives. For example, "But while the two 
Slavic neighbours see themselves as natural partners, 
their relations since the breakup of the Soviet Union 
have been bedeviled" . As two states described in the 
same sentences are likely to temporally overlap and 
any events or times outside or bounding these states 
will be related to the state, it is unlikely that any contri- 
bution to TLINK annotation would be made by linking 
the two states with a "roughly simultaneous" relation; 
the closest suitable label is TempEval's OVERLAP re- 
lation (Verhag en et al., 2010] ). 



(2) "nor can the government easily back down on 
promised protection for a privatized company 
while it proceeds with ..." 

The cases of while that were not of this sense were 
easier to annotate. Sometimes it was used as a tempo- 
ral expression; "for a while" . Other times, it was not 
used in a contrastive sense, but instead as irrealis - see 



Example 12 The four cases of non-contrastive usage 
were annotated as temporal signals. 

Before Three of the ten negative examples are cor- 
rectly annotated. They are before in the spatial sense 
of "in front of (as in "The procedures are to go before 
the Security Council next week") and also a logical be- 
fore that does not link instantiated or specific events 
("before taxes"). The remaining seven unannotated 
examples of the word are all temporal signals. These 
directly precede either an NP describing a nominalised 
event, or directly precede a subordinate clause (e.g. 
[IN before, S] - see Figured). 

Until All eleven non-annotated instances of until 
should have been annotated as temporal signals. This 
word suggests a TimeML IBEFORE relation, unless 
qualified otherwise by something like "not until" or 
"at least until". 

Already There were thirteen positive examples of 
already. All of the non-annotated examples had a non- 
temporal sense as per our description of temporal sig- 
nals. The word tends to be used for emphasis, but can 
also suggest a broad "BEFORE DCT" position, which 
goes without saying for any past and present tensed 
events. As already can be removed without changing 
the temporal links present in a sentence, we have not 
annotated any more examples of this beyond the thir- 
teen present in TimeBank. 

Meanwhile This word tends to refer to a reference 
or event time introduced earlier in discourse, often 
from the same sentence. As well as a temporal 
sense, it can have a contrastive "despite"-like mean- 
ing. Meanwhile tends to refer more to previous ac- 
tions, instead of states specified in immediately prior 
sentences. Sometimes meanwhile is used with no pre- 
vious temporal reference. In these cases, the implicit 
argument is DCT. Five of the ten non-annotated mean- 
whiles were temporal signals. 



Again This word shows recurrence and is always 
used for this purpose where it occurs in TimeBank 
not annotated as a temporal signal. No instances of 
"again " were annotated. 

Former This word indicates a state that persisted 
before DCT or current speech time and has now fin- 
ished. Generally the construction that is found is an 
NP, which contains an optional determiner, followed 
by former and then a substituent NP which may be 
annotated as an EVENT of class STATE. This configu- 
ration suggests a TLINK that places the event BEFORE 
the state's utterance. 

(3) "The San Francisco sewage plant was named in 
honour of former President Bush." 

In Example there is a STATE-class event - Presi- 
dent - that at one time has applied to the named entity 
Bush. The signal expression former indicates that this 
state terminated BEFORE the time of the sentence's ut- 
terance. 

Three-quarters of the non-annotated instances of for- 
mer in TimeBank are temporal signals. 

Recently Although recently is a temporal adverb, it 
can only be to applied simple or anterior tensed verbs 
(using Reichenbach's tense nomenclature). In the cor- 
pus, these are only seen in reported speech or of ver- 
bal events that happened before DCT. Recently adds a 
qualitative distance between event and utterance time, 
but is of reduced use when we can already use tense in- 
formation. The phrase "Until recently" appears awk- 
ward when cast as a temporal signal but can be inter- 
preted as "BEFORE DCT", with the interval's endpoint 
being close to DCT. In this case, recently functions 
as a temporal expression, not a signal. Only one of 
the non-annotated recently?, in TimeBank is a tempo- 
ral signals. The exception, "More recently", includes 
a comparative and is annotated as a T I ME X 3 ; both this 
phrase and, e.g., "less recently" suggest a relation to 
a previously-mentioned (and in-focus) past event. As 
a result, we posit that recently behaves as an abstract 
temporal point (as seen in the behaviour of "until re- 
cently"). Structures such as [comparative] recently 
may be interpreted as a qualified temporal signal, as 
they convey information about the relative ordering of 
the event that they dominate vent compared with a pre- 
viously mentioned interval. 

5. Conclusion 

We have provided a characterisation of temporal sig- 
nal expressions. In an analysis of the TimeBank 
corpus we have shown what proportion of tempo- 
ral relations are explicitly signalled by these expres- 
sions and have given quantitative descriptions of how 



ambiguous these phrases are, both regarding their 
temporal/non-temporal senses and the type of tempo- 
ral relation that they convey. 
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